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1 . 


Introduction 


This report examines the contribution of self-evaluation to internal review processes 
in Scotland’s colleges. It identifies the range of methodologies used by colleges, 
what makes these effective and what needs to improve further. It explores in 
particular, the extent to which self-evaluation has a positive impact on learners’ 
experiences. The report addresses these aims through: 

• evaluating the extent to which self-evaluation has improved over time, including 
taking into account equality and diversity issues; 

• evaluating the contribution made by external stakeholders and college learners 
as part of the self-evaluation process; 

• identifying and disseminating examples of excellence or sector-leading and 
innovative practice; and 

• making recommendations for improvement to current practice. 

Colleges have a relatively long history of self-evaluation since they became 
independent corporations in 1993. Almost 20 years on, all colleges have developed 
quality assurance, improvement and enhancement systems for teaching and cross- 
college support areas based, in almost all cases, on self-evaluation of their provision. 
They have implemented formal reporting arrangements for service areas and 
curriculum teams which help them gauge performance and monitor progress. 

The HM Inspectorate of Education (HMIE) report: Effective self-evaluation reporting 
in Scotland’s colleges (2007), ] identified a number of positive features including that: 

• self-evaluation activities and reporting is well embedded in quality processes; 

• almost all staff are familiar with and have actively engaged with reporting 
processes; 

• self-evaluation has a positive impact overall on improvement in services to 
learners. It has encouraged a culture of reflection and questioning amongst 
college staff and underpins a focus on continuous improvement; 

• the quality of self-evaluation reports has generally improved over the last four 
years; 

• staff have a positive attitude towards self-evaluative activities; and 

• self-evaluative reporting is most effective when it builds on and reports the 
findings and impact of professional dialogue and other self-evaluative activities. 

However, there were a number of recommendations for improvement: 

• colleges should encourage and facilitate self-evaluation activities such as 
professional dialogue, especially for learning and teaching, retention and 
attainment; 

• reports should record the impact of previous actions; 

• reporting should focus on learning and teaching, retention and attainment and 
involve rigorous analysis of evidence including performance indicators (Pis); 

• reports should be evaluative rather than descriptive; 

• quality improvement action plans should contain SMART targets; and 


1 http://www.hmie.qov.uk/documents/publication/esersc.pdf 
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• colleges should provide targeted training and support to staff to improve 
evaluative skills and self-evaluation report writing. 

HMIE has provided other analyses of self-evaluation and internal review 
arrangements in colleges. The most recent was through the HMIE publication, 
Improving Scottish Education 2 (2009) 2 , it commented: 

“All colleges have a strategic commitment to internal review and most staff are 
actively involved in self-evaluation processes. Almost all staff demonstrate good 
levels of knowledge of quality procedures and are committed to improving the 
quality of the learner experience. The self-evaluation and quality improvement 
activities generally lead to enhancement of services for learners. Most colleges 
have well-developed, robust systems for gathering and analysing performance 
data and the views of learners and other stakeholders. However, not all college 
staff make sufficient use of this information to devise or implement effective 
action plans to improve cases of low retention and attainment, especially when 
weaknesses relate to learning and teaching processes. ” 

This report identifies the progress that colleges have made, since the publication of 
these two reports, in improving their approaches to self-evaluation and internal 
review. 


2 http://www.hmie.qov.uk/documents/publication/ise09.pdf 
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2. Methodology 


Eight colleges were visited by two HMI or associate assessors during the fieldwork. 
During the visits a number of interviews were held consisting of: 

• professional discussions with key managers, college staff and relevant external 
stakeholders; 

• discussions with learners and their representatives; 

• discussions with colleges on internal review procedures; and 

• discussions with managers on the arrangements for the evaluation of learner 
progress and outcomes, learning and teaching, learner engagement and quality 
culture. 

Self-evaluation reports and other quality documentation collected for college reviews 
and provided by other colleges were analysed. After the visits, HM Inspectors 
convened a focus group of staff involved in the fieldwork for the report, in order to 
share their experiences and to discuss the findings. The annual reports submitted 
by colleges to the Scottish Funding Council (SFC) were also analysed. In addition to 
the evidence obtained from the eight colleges that participated in the fieldwork stage, 
FIM Inspectors also drew on the evaluations contained in the published external 
review reports of 24 colleges between 2008-1 1 . Additionally, a number of annual 
engagement reports were analysed, where appropriate, in order to determine 
emerging practice relating to self-evaluation and internal review. Evidence from 
these external review activities has helped to identify areas of progress as well as 
highlight issues that continue to require attention. 
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3. Summary of key findings 

Self-evaluation in Scotland’s colleges is characterised by the following strengths: 

• Arrangements for self-evaluation have matured and improved. Colleges now 
have well-established and widely accepted quality assurance and improvement 
practices. 

• Self-evaluation reporting, in most colleges, is a well-developed continuous 
process, with identified timelines and contributions from staff and learners 
spread throughout the year. 

• Colleges have developed a variety of approaches to self-evaluation, using a 
range of quality frameworks which suit their individual needs. 

• Colleges have developed effective arrangements for curriculum review. Where 
new programmes of study have been introduced, these often take good account 
of the needs and interests of learners. 

• Support teams have devised a range of approaches in order to evaluate and 
improve the provision of the services they provide to learners. 

• Colleges value the contributions made by learners. They have introduced a 
range of procedures designed to capture the views of learners. 

• Colleges engage effectively with employers. 

• Increasingly, colleges identify in their self-evaluation reports the approaches they 
are adopting to deliver the outcomes of Curriculum for Excellence. 

However, we noted a number of areas for development to improve the outcomes of 

self-evaluation and internal review: 

• More than a few colleges do not have sufficiently rigorous or robust approaches 
to college-wide self-evaluation which ensures comprehensive and accurate 
self-evaluation outcomes. 

• Too few colleges compare their own PI outcomes with appropriate external 
benchmarks. 

• More than a few programme reports do not identify appropriate actions to 
address improvements for learners by defined equalities categories. Most 
self-evaluation reports omit reference to the promotion of equality and diversity 
through the introduction of refreshed teaching material or adjustments to 
teaching practices. 

• In more than a few colleges, self-evaluation reports contain insufficient reflection 
by staff on things they can influence, including approaches to learning and 
teaching. 
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• In more than a few colleges, learner representative arrangements are 
insufficiently effective. 

• Few colleges have established college-wide review arrangements, with receiving 
Higher Education (HE) providers, in order to share their approaches and to 
review curriculum content. 

• Few colleges have developed formal arrangements with schools to share, 
discuss and ensure effective contributions are made by both parties to joint 
evaluation of school-college programmes. 
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4. College-led internal review processes - what do we mean? 

Internal review is the collective term for a range of college-wide quality assurance 
and enhancement activities. It encompasses a number of interrelated processes 
including: 

• self-evaluation; 

• curriculum review; and 

• internal audit. 

Self-evaluation is the main approach used by colleges for the analysis and reporting 
of their strengths and areas for development and as such, provides the primary 
subject for this report. Self-evaluation helps to provide quality assurance, but it also 
helps to identify what needs to improve or be enhanced and indicates what actions 
can address these. In identifying how well they are doing, colleges usually gather 
evidence from a number of sources including: 

• analysis of the views of learners and other stakeholders; 

• observations of learning and teaching; 

• findings from external and internal moderation reports; 

• analysis of Pis relating to learner retention, achievement, attainment and 
progression; and 

• reflections gathered by individuals and members of teams. 

At times, national benchmarking information is also used in self-evaluation. 

Over time in all colleges, self-evaluation has become increasingly embedded in 
everyday practice. Many colleges now view it as a continuous process with 
contributions from staff and learners spread throughout the year. This complements 
the formal and cyclical quality reporting processes which also feature in all colleges. 
Most staff take advantage of a number of opportunities to reflect on what has gone 
well and what needs to be done to improve further. These are often informal and 
based around everyday professional dialogue between colleagues, but also include 
more methodical, formal reflection against a quality framework, carried out at specific 
times of the year. Through the self-evaluation process, staff identify strengths, areas 
for improvement and examples of good practice for wider dissemination across the 
college. This process culminates in plans for improvement, examples include: 

• improved learner retention and attainment rates; 

• improvements in learning and teaching approaches; and 

• better access to information and communications technology (ICT) 

• and enhanced services to learners. 

Curriculum review is the process of analysing, reviewing and where appropriate 
refreshing or terminating particular programmes which are delivered within a college. 
Through these arrangements, colleges ensure that the portfolio of programmes 
meets the needs, interests and aspirations of learners as well as responding to 
national initiatives such as learners in need of more choices and more chances. 
Usually, the criteria for the implementation of curriculum review are low or declining 
learner recruitment, retention or attainment. 
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Internal audit includes a range of activities, usually carried out by college managers. 
Examples include: 


• internal verification of qualification standards; 

• analysis of quality of learning and teaching materials, including their focus on 
equality; 

• reviews of support services such as admissions, ICT and Human Resources 
(HR); 

• identification of implementation of Curriculum for Excellence; and 

• how well liaison with employers is impacting on learning. 

On occasions, internal audits are scheduled when colleges determine that there are 
specific themes which require further investigation or improvement. Many colleges 
implement internal audits annually selecting one or two departments or service 
areas. 


Ayr College: internal review of bursary arrangements 

As part of the internal review of bursary arrangements, managers carried out an 
internal audit of the practices being used over a two week period in April and May 
20 1 0. From discussions held with staff and learners, they identified that a number 
of learners submitted incomplete bursary applications and some administrative 
procedures were burdensome. As a consequence, a number of learners 
experienced delays in receiving their bursary. Through the college-devised 
internal review, new improved arrangements for bursary information, processing 
and allocation were agreed, resulting in a much improved service for learners 
commencing programmes in 2010-11. Through this internal review, Ayr College 
has identified the root cause of issues which directly affect learners. They have 
put in place new improved arrangements which ensure that the learner experience 
at the college continues to improve. 
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5. Approaches to self-evaluation in Scotland’s colleges 

• what works well? 

• what does not work so well? 

Self-evaluation practices 

Most college-led internal review activity is based on self-evaluation. Self-evaluation 
is carried out extensively by managers and teaching or support staff teams who 
evaluate their evidence against a quality framework. These reports contain actions 
for improvement. 

All colleges now have well-established processes and procedures which enable 
them to conduct effective self-evaluation and internal reviews for both teaching and 
cross-college support areas. Significant improvements have been made across the 
sector overall, in order to ensure that self-evaluation procedures are now less 
burdensome, focus on the key issues and identify progress being made over time. 
Self-evaluation arrangements are now, in most colleges, part of a continuous 
process governed by the needs of the college, rather than by external requirements. 
They are well embedded and have a positive impact overall on improvement in 
services to learners. 

Since the publication of Effective self-evaluation reporting in Scotland’s colleges 
(2007), there has been a noticeable improvement in approaches to self-evaluation 
within most colleges. In the best colleges, self-evaluation reporting is characterised 
by: 

• comprehensive and detailed reports; 

• evaluative reports which contain SMART targets; 

• aggregated whole-college reports which draw clearly on those of curriculum and 
support areas; and 

• staff empowered to draw up targets for improvement and held accountable for 
their actions. 

However, in more than a few colleges, self-evaluation is not always carried out or 
reported so well. In these colleges, self-evaluation reports are sometimes 
insufficiently evaluative, do not analyse evidence rigorously enough and often fail to 
set specific actions and targets for improvement. As a result, self-evaluation in more 
than a few colleges does not have the impact on improvement that it could. These 
colleges are unable to provide assurance, through their self-evaluation reports, that 
teams are continuously improving services for learners. 

These less effective self-evaluation reports often include: 

• action plans containing targets which are not specific or measurable; 

• few references to teaching practices or approaches; 

• insufficiently robust evaluation of identified weaknesses; and 

• inconsistent analysis of collated reports with no clear link to college-wide 
improvement planning processes. 

Colleges use a variety of quality frameworks against which they evaluate their 
provision. Most colleges have drawn up or adapted frameworks which are 
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appropriate for their own needs, and help staff to focus on essential aspects of 
services for learners. Most colleges currently base their self-evaluation on the latest 
HMIE framework for external quality review, External quality arrangements for 
Scotland’s colleges (2008) 3 . This framework was introduced after extensive 
consultation and has been well received by the sector, overall. It reflects the shifting 
focus of external reviews and the recognition of the evolving importance of the role of 
learners through four high-level questions: 

• how well are learners progressing and achieving relevant, high quality 
outcomes? 

• how effective are the college’s learning and teaching processes? 

• how well are learners engaged in enhancing their own learning and the work and 
the life of the college? 

• how well is the college led and how well is it enhancing the quality of its services 
for learners and other stakeholders? 

Some colleges have introduced additional criteria for their own purposes. For 
example, a few colleges incorporate aspects of the previous version of the HMIE 
framework for external quality review (2004-08), which included grading on a four 
point scale: 

Very good - major strengths 
Good - strengths outweigh weaknesses 
Fair - some important weaknesses 
Unsatisfactory - major weaknesses. 

They particularly value the opportunity to construct a grade profile for curriculum and 
support areas. This helps them to monitor how well they are performing over time 
and what aspects of their provision requires further improvement. 

A number of colleges are investigating or are using the European Framework for 
Quality Management { EFQM) 4 excellence model offered through Quality Scotland. 
This self-evaluation model has recently been developed to meet the needs of 
educational providers. Some colleges have successfully drawn up their own quality 
frameworks through consultation with staff. 


3 http://www.hmie.qov.uk 

4 http://www.qualitvscotland.co.uk/efqm.asp 
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Kilmarnock College: Sector leading practice in developing The Wheel 

During session 2009-10 the college revised its learning, teaching and assessment 
strategy. As part of this revision, a visual representation of the strategy was 
developed - ‘The Wheel’. This wheel is divided into segments representing the six 
guiding principles of the strategy: 

Innovative 

Flexible 

Supportive 

Reflective 

Successful 

Engaging 

Each segment details relevant and specific criteria which are used as a basis for 
the self-evaluation process. This criteria relates to the external influences from all 
stakeholders including Curriculum for Excellence, Skills for Scotland and HMIE. 
Four quality indicator (Ql) statements contribute to each outcome. Staff evaluate 
their progress against each Ql and attribute a score out of ten for each, 
culminating in a score out of 40 for each outcome. This model was so successful 
that it was adapted and adopted by all curriculum and service areas. 

Without exception, all teams have felt greater ownership of self-evaluation and 
have engaged in professional dialogue with colleagues focusing in on the critical 
detail of their area. Having a model in place, where staff have been engaged in 
the development of the model, has helped understanding, ownership and 
commitment. 

As a visual tool the wheel has enabled all teams to have clear agreed actions for 
improvement for 2010-11. A model for learners to encourage structured learner 
feedback has now been developed through a learner focus group and learners are 
extremely positive about this new approach. 


Colleges receive an external perspective on their self-evaluation, curriculum review 
and internal audit activities through the review activities of HMIE. The HMIE review 
framework contains quality indicators which include: 

• how well the college sustains continuous enhancement through self-evaluation 
and internal review activities? 

• how well does the quality culture in the college lead to quality improvement and 
enhancement? 

External reviews provide independent validation of the impact and success of 
particular developments undertaken by colleges, and of their quality assurance and 
improvement arrangements. They can also provide a prompt for change where 
issues are identified for improvement. Colleges also receive on-going support from 
their link HMI on a range of issues which are agreed with each college. These often 
include support relating to their approaches to self-evaluation. 


10 




Support service teams in colleges carry out self-evaluation using a range of 
frameworks. Most use aspects of the current HMIE review model alongside 
internally developed criteria. There is recognition by many college managers that a 
more bespoke process is required by support service areas in order to gather 
relevant feedback from service users. Almost all colleges have developed 
appropriate self-evaluation templates for completion by support service personnel. 
College managers agree that the primary contributor to the success of the 
self-evaluation activity is not the particular framework used, rather it is the 
commitment and full involvement of staff at all levels in the organisation. While there 
are still areas for improvement in more than a few colleges, staff are committed to 
improvement in services for learners. Self-evaluation is now much more integrated 
into college quality culture overall, and is rarely seen simply as a paper exercise 
completed for bureaucratic purposes. 

In teaching teams in all colleges, self-evaluation focuses on the performance of 
programmes which colleges offer to learners. Through the analysis of learner 
retention and attainment, staff can identify which programmes have high Pis and 
which ones require improvement. Most colleges have developed sophisticated and 
detailed analysis of Pis in order to identify how well they are performing. 

A number of colleges use a traffic light system of green/amber/red in order to identify 
high/medium/low performing programme Pis. These are often used for the 
categories of early retention, retention and student outcomes. This approach 
enables teaching staff and college managers to identify quickly high and low 
performing programmes when reviewing PI outcomes. Where programmes have a 
red indicator, programme teams are expected to draw up an improvement plan as a 
matter of urgency, with SMART targets for improvement relating to learner retention 
or outcomes. Where staff respond and make significant changes to either 
programme design or delivery, these actions often produce improved outcomes for 
learners. However, in more than a few colleges, a number of self-evaluation reports 
contain insufficient reflection by staff on aspects they can influence. For example, 
these reports fail to consider approaches to learning and teaching, and often 
attribute low PI outcomes to vaguely defined learner issues, such as dropped out or 
personal circumstances. Where there is little change to programme design, entry 
criteria or delivery approaches, negative PI patterns are often repeated the following 
year. 
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Angus College: effective approaches to PI evaluation 

Academic teams complete detailed Annual Course Reports (ACR), for each of the 
college’s 105 mainstream programmes. These evaluate the learner experience 
and set targets for improvement. Each ACR is evaluated by the Directors of 
Learning and Teaching and a feedback sheet is completed and discussed with 
staff using the following traffic lighting system: 

GREEN - these figures are set high to reflect the aspi rational target of the college 
strategic plan; 

AMBER - these figures generally reflect the national standard for successful 
outcome; 

RED - these figures show areas of concern but should also be measured against 
the national benchmark for the given subject area. 


Data categories 

SRR1 (retention 
rate for first 25% 
of programme) 

90-100% 

75-89% 

74% or lower 

SRR2 (retention 
rate for full 
programme) 

85-100% 

75-84% 

74% or lower 

SARU (unit 

achievement 

ratio) 

85-100% 

75-84% 

74% or lower 

PSO (positive 

student 

outcome) 

85-100% 

75-84% 

74% or lower 

SCR (successful 
completion ratio) 

85-100% 

75-84% 

74% or lower 


The use by the college of this comprehensive range of data categories, allied to 
an easily recognisable colour coding system, provides a detailed and transparent 
feedback process to programme teams. The traffic light boundaries are higher 
than those used by most colleges. College managers also provide feedback to 
staff through the use of confidence statements. These confirm to staff the 
confidence held in respect of the appropriateness of their evidence base and the 
quality of evaluation used to inform their reports. 

Confident (C)- staff demonstrate appropriate evaluative skills, actions and 
evidence base. 

Confidence with caveats (CC) - staff are required to improve their evaluations 
and to improve their evidence base. 

Not confident (NC) - staff fail to demonstrate confidence through their 
self-evaluations and action plans. 

These comprehensive and detailed reporting arrangements, combined with 
informative feedback from their line managers, ensure that staff at Angus College 
produce high quality, accurate, and enhancement focused self-evaluation reports. 
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In order to improve access by staff to real-time Pis, many colleges have introduced 
dedicated software such as the Enquirer system. This is helpful to college personnel 
and enables them to closely monitor live attendance and progression data for 
learners. This ensures that accurate data is being made available for staff to use 
and to share with their colleagues when discussing progress being made by 
individuals and groups of learners. 

When evaluating programme outcomes, teaching teams take cognisance of their 
own PI values and their improvement or decline over time. On occasions, teaching 
teams benchmark their own programme Pis against those of similar programmes. 
Where external benchmarking occurs, programme teams sometimes use sector 
average values. This approach can lead programme teams to conclude that their PI 
outcomes are acceptable as they match the average for the sector. Sector averages 
are by definition a mixture of high, average and low performing outcomes, and they 
also represent very simplistic aggregations of very different types of programmes 
across all colleges. Because they have remained static for full-time Further 
Education (FE) and HE programmes across Scotland for the last three years, using 
these benchmarks will not necessarily produce aspirational action plans and targets 
for improvement. 


Sector 

outcome 5 


07/08 

08/09 

09/10 

FE 

82 

82 

82 

HE 

82 

82 

81 


The benchmarks used by Angus College, in the previous example, reflects the 
college’s ambition to be Scotland’s best community college. Few colleges 
benchmark their own performance using college-devised benchmarks or against 
high performing colleges with a similar learner profile. 


Aberdeen College: self-evaluation guidance notes for programme teams 

Guidance notes make clear to staff the sources of evidence to be used when 
compiling self-evaluation reports. PI data is readily available to programme 
teams. The responses to learner surveys are collated and colour coded. This 
supports staff to quickly identify which questions have a positive response and 
which ones need further investigation and potentially should be included in the 
team’s self-evaluation action plan for improvement. 


Role of the Board of Management and senior managers 

Since 2008, Boards of Management (BoM) have been given increased responsibility 
for monitoring and assuring the quality of provision. Colleges have a responsibility, 
through their BoM, to produce an annual report which describes the scope, nature 
and outcomes of college-led quality review activities. These reports are also 
intended to include commentary on actions taken to bring about improvement. In all 
colleges, Board members are aware of their responsibilities and are fully committed 


5 http://www.sfc.ac.uk/statistics/facts fiqures/091 0/091 0.aspx 
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to quality enhancement within their own institutions. However, Boards differ in their 
interpretation of the level of detail which they deem necessary to provide in their 
annual report to the SFC. Some are very detailed and analyse progress made 
against previous targets, and often include reference to learner outcomes alongside 
fresh targets for the coming year. Other reports are less helpful. These often focus 
on internal processes and procedures and fail to make clear reference to learner 
progress and outcomes. The guidance from SFC is for annual reports to be no more 
than three or four pages. Many reports provided by colleges exceed ten pages with 
the longest, including appendices being 73 pages long. From discussions held with 
colleges, a number of them question the value of producing this report. They view it 
as a paper exercise which does not change significantly in content or substance 
each year. 

Outcomes and targets for improvement contained within college self-evaluation 
reports are discussed with college staff by HM inspectors during external reviews. 
More than a few published HMIE external review reports contain reference to areas 
for improvement, either within the body of the report or as main points for action, 
which had not previously been identified or recognised by these colleges as areas 
for improvement. This indicates that these colleges do not have sufficiently rigorous 
or robust approaches to self-evaluation, either in particular teaching or support 
departments, or in whole-college self-evaluation which ensures comprehensive and 
accurate self-evaluation outcomes. There is a danger that these colleges and their 
BoM have an overly optimistic view of their own performance and a belief that they 
do not have any main areas for development. As a result, targets for improvements 
are insufficiently challenging and do not ensure improved outcomes for learners. 

In most colleges, principals and senior managers play an active and useful role in 
self-evaluation. Many chair meetings with managers and staff, and discuss regularly 
the impact on learners of their departmental self-evaluation reports and other internal 
review activities. Through these arrangements, managers are kept informed of 
relevant developments across the college. Staff explain the progress they have 
made in their own departments and the impact of improvement plans against last 
year’s targets. Staff find these meetings useful. They feel that their viewpoint is 
valued and their contributions are taken seriously. Many principals actively strive to 
hear the views of learners. They hold focus groups or informal meetings with 
learners at a range of times and venues, including the college refectory. They hear 
for themselves the issues concerning learners and often take corrective action or 
raise issues with colleagues in order to bring about improvement. 
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Coatbridge College: principal’s formal review 

Curriculum and support managers are invited to meet with their respective line 
manager, quality manager and the principal during the principal's formal review. 
During these meetings, managers deliver a presentation based on the outcome of 
their own department or curriculum area self-evaluation and the resulting 
confidence statement. The quality unit supports this process and provides a 
bespoke template distributed in advance to managers, using HMIE confidence 
statements, with a concluding slide identifying plans for improvement. The 
presentation highlights strengths, areas for improvement and any good practice to 
be shared across the college. All proposed confidence statements are supported 
by relevant evidence. During these presentations, the principal and quality 
manager question the presented evidence base in order to determine the validity 
and accuracy of the proposed confidence outcome. Through this supportive yet 
challenging process, curriculum and support managers demonstrate knowledge of 
their own departments and what they need to do in order to improve. This 
detailed scrutiny of practices and progress is welcomed by managers. They value 
the opportunity to share and discuss with the principal and the quality manager 
the key issues which impact on the day to day running of their own departments. 


Collaboration with other colleges 

Teaching staff whose higher education programmes are affiliated to the University of 
the Highlands and Islands (UHI), often implement quality assurance, improvement 
and enhancement processes through collaboration with colleagues from other UHI 
institutions. Overall, this process works well. Staff often discuss their self-evaluation 
outcomes with each other and collaborate in order to produce relevant and effective 
self-evaluation reports. Higher education programmes are often developed and 
delivered by teaching staff across a number of UHI colleges. Staff value the 
opportunity to share their thoughts and to seek agreement on what works well and 
what needs to improve further. These collaborative arrangements do not, however, 
extend to FE programmes delivered in UHI colleges, as these are the responsibility 
of individual institutions. More widely across Scotland, few colleges have developed 
formal arrangements to share their approaches to self-evaluation with their peers in 
other colleges. Many college managers benefit from the support provided by 
Scotland's Colleges through their well-established Communities of Practice. Quality 
managers meet regularly and discuss a number of quality-related issues, including 
approaches to self-evaluation. 

Learner contributions 

All colleges demonstrate a commitment to enable learners to contribute effectively to 
self-evaluation. The revised HMIE framework of 2008 introduced the criteria of 
learner engagement as one of the four key principles of external review. This has 
prompted colleges to re-examine their approaches to learner representation and 
consultation. The practice of listening to, recording and acting on the views of 
learners has developed and improved significantly over recent years. All colleges 
have learner representation systems where learners elect class representatives to 
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listen to their views, and to present these either at formal programme team meetings 
or informally throughout the year. 

Colleges use a number of surveys, normally distributed electronically, after induction, 
mid-way through the academic year and towards the end of programme, to gather 
the views of learners about their experiences. Many programme teams make 
effective use of these surveys in their seif-evaluation reports in order to identify what 
is going well and what requires further improvement. Staff often distribute end-of- 
unit evaluations and receive helpful feedback from learners about the subjects they 
are studying. Through these arrangements, learners play an active role in 
contributing to programme team self-evaluation reports. However, few programme 
teams provide opportunities for learners to read and comment on the outcomes of 
self-evaluation reporting. 


Ayr College: learner involvement in internal review 

As part of their learner engagement strategy, Ayr College includes learners as 
members of their internal review teams. Learners are invited to apply annually for 
these remunerated posts. Applicants are interviewed and undertake a Disclosure 
Scotland application. During the week-long internal reviews, learners take a full and 
active part in the internal review process. All members of the internal review team 
discuss and agree the final report which is presented to the college’s Curriculum and 
Quality Steering Group. 


Employer contributions 

All colleges demonstrate a commitment to seek out the views of employers. They 
understand the importance of establishing and maintaining effective links with local 
industries to help college learners progress into employment at the end of their 
programmes. All colleges take good account of a wide range of external drivers, 
including Scottish Government initiatives and policies, sector skills council reports 
and the views of other stakeholders, including local and in some colleges national 
employers. Often, colleges interrogate local and national labour market statistics in 
order to identify existing and future demands for their programmes. 

Colleges have developed a range of ways of gathering the views of employers. 

These include: 

• discussions held with college staff; 

• through work-placements offered to learners; 

• input by employers to specific topics or programme content during teaching 
sessions; 

• visits to employer premises by learners; and 

• representation by employers as part of formal college meetings. 

A number of colleges hold successful employer engagement events where they 
invite representatives from local industries for business meetings. College managers 
provide updates on recent developments and discuss with employers potential 
opportunities for further collaboration. Colleges and most programme teams make 
effective use of the active support and input they receive from employers. This helps 
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to keep them informed about a range of new initiatives, including any relevant 
changes to industrial practices. In their self-evaluation reports, many programme 
teams identify how employers make effective contributions to their programmes and 
what has changed as a result. However, few colleges formally share with employers 
their self-evaluation outcomes. 

Other external stakeholders or influencers 

Almost all of the colleges in Scotland have strong and effective working 
arrangements with a wide range of external partners. These include local authorities 
(in some colleges this amounts to more than one), schools, local health boards, HE 
providers and other community partners. Often they work closely together to ensure 
that they are meeting the needs of groups of learners such as learners in need of 
more choices and more chances, migrant workers, adult returners and unemployed 
or redundant workers. Working arrangements have been well-established over a 
number of years. Some subject areas have close, informal networks with receiving 
HE institutions. There are examples where individual members of staff share their 
curriculum, teaching and assessment practices which helps to ensure that learners 
are appropriately prepared for progression onto university programmes. However, 
these are the exceptions. Programme teams often develop and deliver programmes 
to learners with HE aspirations, without the active support and advice of their 
university colleagues. Enthusiastic members of staff work diligently to ensure that 
their learners can progress onto receiving HE institutions. However, few colleges 
have established college-wide review arrangements, with receiving HE providers, in 
order to share their approaches and to review curriculum content. Too often this 
only occurs through the endeavours of committed, individual members of staff. 

Most colleges have established effective working relations with their local schools 
through their school-college partnership provision. Some colleges engage with over 
1000 school pupils each week studying a wide range of vocational and academic 
programmes. School pupils normally attend college one morning or afternoon each 
week, so there are limited opportunities for these learners to actively contribute to 
programme team meetings. In many colleges, staff have devised a number of 
effective ways to gather the views of school pupils through the distribution of 
questionnaires. These often have high completion rates and inform enhancements 
in approaches to learning, teaching and assessment. While most college staff inform 
their school colleagues about the progress their learners are making and the 
qualifications which they achieve, few colleges have developed formal arrangements 
for joint evaluation of school-college programmes. Communication generally 
focuses on learner issues such as attendance, behaviour and attainment of units. 

Equality, diversity and sustainability 

From a slow start in previous years, an increasing number of colleges now produce 
data on learner retention and attainment by a range of protected characteristics, 
most commonly gender and disability. Aggregated college-wide data is used to 
inform the college’s annual equality and diversity reports. However, too often, this 
data is not dis-aggregated in a format which allows programme teams to analyse 
learner recruitment, retention and attainment for their own programmes. Many 
teaching staff are therefore unable to analyse in their self-evaluation reports, the 
progress being made by particular groups of learners. As a result, more than a few 
of these reports at programme level fail to include action plans which address 
improvements for learners by defined equalities categories. 
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Colleges have a responsibility to ensure that their teaching materials promote 
equality and diversity and are free from bias. Almost all colleges have adopted the 
Quality and Equality in Learning and Teaching Materials (QELTM) toolkit, to ensure 
teaching materials promote diversity through their content and delivery approaches, 
as well as being available in accessible formats for learners with additional needs. 
However, there is a need for staff to embed the promotion of equality and diversity 
further into their delivery and teaching practices rather than regarding QELTM as the 
totality of their promotional efforts. In colleges where this works well, self-evaluation 
reports include an evaluation of promotion across the whole programme as well as 
teaching methods and approaches. However, few reports include commentary by 
teaching staff on equality and diversity approaches in their teaching practice. 

Overall, Scotland’s colleges have responded well to the Scottish Government’s 
Greener Scotland 6 strategic objective. Almost all colleges embed sustainability 
within their strategic plan, aims and objectives and the majority have clear policies 
and strategies to address sustainability in most aspects of their operation. 
Increasingly, college staff and learners display good awareness of sustainability 
issues and their significance to both the college and the wider community. However, 
few programme team self-evaluation reports contain reference to actions taken in 
order to support achievement of this aspect of the college’s strategic plan. In most 
colleges, responsibility for reporting on progress being made on sustainability issues 
remains the responsibility of a nominated senior manager. 


6 http://www.qoqreenerscotland.org/ 
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6. Impact of internal review on the learner experience 
High quality learning 
Evaluating learning and teaching 

Colleges have developed a range of processes which aim to assure and improve the 
quality of learning and teaching approaches. These can include: 

• formal observations of learning and teaching activities, by managers or peers; 

• meeting forums where practitioners discuss their practice; 

• best practice events, where teaching staff share and demonstrate their 
approaches; 

• continuing professional development (CPD) events, where all teaching staff 
receive updates and development relevant to their roles as teachers; 

• peer observation where colleagues observe teaching sessions and provide 
feedback, support and suggestions for improvement; and 

• feedback received from learners through focus groups, meetings and 
questionnaires. 

Colleges which carry out observations of learning and teaching use a wide variety of 
approaches. Some colleges observe all staff, including part-time staff annually. 

Other colleges spread this across a two or three-year cycle. One college has an 
observation cycle to ensure all staff are observed over a four-year period. A number 
of colleges are aware of the benefits of having a college-wide observation process 
but are either at the early stages or have not yet made significant progress in this 
area. Teaching staff in colleges which carry out observation of learning and teaching 
usually value the process. They particularly benefit from the subsequent discussion 
and in many cases they respond by introducing new initiatives and approaches in 
their teaching. In the most recent submissions by colleges on institution-led review 
to the SFC, around 30% of colleges report that a system of observation of teaching 
practice is in place, is being implemented or is planned. A significant number of 
colleges, therefore, have no observation protocols. As a result, teaching staff do not 
receive direct professional feedback on the quality of their teaching. 

A few colleges attribute improvements or enhancements in learning and teaching 
directly to the observation process. While this may contribute to more accurate 
evaluation of learning and teaching, many colleges still find it difficult to identify 
clearly what needs to improve in these areas, whether they operate classroom 
observation or not. Evaluation of learning and teaching has not significantly 
improved since it was highlighted in HMIE’s report of 2007. As a result, many 
colleges still find it difficult to identify appropriate and clear action plans to improve 
learning and teaching. 

Where learning and teaching is reported as a main point for action in external 
reviews, this often relates to the evaluation of learning and teaching by programme 
teams. Overall, learning and teaching practice is effective. In these colleges, 
individual members of staff identify for themselves teaching approaches which meet 
the needs of learners. However, there is often no collective analysis of these 
practices to inform wider improvement in approaches to learning and teaching by 
programme teams. 
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Colleges use other means to evaluate learning and teaching. For example, many 
colleges find it helpful to take into account the views of learners as the primary 
source of evidence in order to make judgements on the overall quality of their 
approaches to learning and teaching. This can provide very clear and helpful 
indicators of what needs to improve, from the customer perspective. Almost all 
teaching staff also discuss their approaches to learning and teaching through 
informal interactions with colleagues and in the professional dialogue that supports 
more formal meetings. However, despite the assertion by many teaching staff that 
learning and teaching is discussed fully when colleagues meet, most teaching team 
meeting notes reflect discussions focused primarily on operational issues and only 
rarely focus on learning and teaching. Increasingly, however, self-evaluation reports 
do contain reference to Curriculum for Excellence. Staff comment on the 
approaches they have taken in order to develop for their learners the four capacities 
of successful learners, confident individuals, responsible citizens and effective 
contributors. These comments often focus on how learning and teaching are helping 
learners develop the four capacities. Through this approach, and increasing levels 
of awareness, programme teams demonstrate confidence and understanding of how 
Curriculum for Excellence can enhance the learner experience. 


Angus College: lesson observation arrangements 

The formal observation process has a developmental focus. Observations are not 
part of individual's annual performance review. As a result, staff feel that this 
supports them to be innovative during their planned observations. Prior to an 
observation, staff are encouraged to consider and try new teaching initiatives 
which they have not tried before. They use observation experiences in order to 
further develop their own teaching skills and techniques. The professional 
discussion held with their observer is helpful and supportive. Staff feel that the 
lesson observation arrangements at Angus College help them to become better 
teachers. 


Ayr College: lesson observation arrangements 

A revised lesson observation process was introduced in 2010-11. Curriculum 
Leaders (CLs) form a team of ten reviewers who observe lessons across the 
college. Each undertakes 20 observations and all 200 teaching staff are reviewed 
annually. Observation outcomes are passed onto the CL of the staff member 
observed with identified strengths and areas for development. Where innovative 
practice has been observed, this is identified for cascading more widely across the 
college. A Learning and Teaching Forum has recently been convened to consider 
key messages from the observation process and take forward college-wide 
actions for improvement. 
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Aberdeen College: lesson observation arrangements 

The college has developed wide ranging and comprehensive arrangements to 
ensure all teaching staff benefit from lesson observations. Teaching staff can 
receive an unannounced observation at any time during the year. These 
observations are normally conducted by an individual’s line manager and this is 
part of the annual performance review process. Teaching staff feel that their 
practice benefits from these arrangements. Additional support for teaching staff is 
provided through the use of peer observations. Large numbers of staff benefit 
from this service. Newly appointed teachers are assigned a mentor who supports 
them through the early stages of their teaching career. They are provided with 
opportunities to observe teaching sessions outside of their own curriculum areas. 
For example, English for Speakers of Other Languages (ESOL) staff conducted 
an observation on Sport and Leisure programmes in order to identify different 
teaching approaches. Staff value and benefit from these experiences. 


Learner engagement 

Almost all learners consider that they have positive relationships with teaching staff 
and that teaching staff take good account of their views informally during classes, 
outwith the learner representative system. Learner representatives generally carry 
out their duties effectively. Those who have benefited from training, either by 
student participation in quality Scotland (sparqs) 7 , or college-devised training, are 
usually well prepared for their roles. They are more confident about their 
responsibilities and often have high attendance rates at programme team meetings 
or provide insightful comments gathered from their classmates before meetings. 
Where learner representatives contribute constructively to programme team 
meetings, self-evaluation reports often include reference to changes and 
enhancements made as a result of feedback received from learners. However, this 
good practice is less well established in more than a few colleges. In these 
instances, large numbers of representatives remain untrained and unprepared for 
their role. More than a few self-evaluation reports make reference to poor 
attendance or ineffective contribution by learners at programme team meetings. 
These reports frequently fail to identify the causes of this low attendance and in 
some cases this is a repeating pattern from year to year. 

Despite variable contributions by learners to programme team meetings, learners’ 
views are often influential in bringing about change for the better. For example, in a 
number of colleges, learners express concern over the scheduling of their 
assessments or the nature and running order of specific subject units. In response, 
programme teams make adjustments in order to improve the experiences of next 
year’s cohort. Colleges are also becoming increasingly effective at providing 
feedback to learners who have raised concerns. This is particularly evident in public 
and social areas of colleges, for example through You Said We Did messages on 
posters or plasma screens. In these ways, college staff demonstrate to learners that 
their views are taken seriously and are acted upon. 


7 http://www.sparqs.ac.uk/ 
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Ayr College: arrangements to ensure attendance by learners at programme 
review meetings 

For learner views to have an influence on learning, teaching and assessment 
approaches, there has to be high levels of attendance by representatives at these 
meetings. Ayr College has developed a protocol that programme team meetings 
can only be held if the learner representative is present at the meeting. Through 
this arrangement, the college ensures that there are consistently high levels of 
learner representation at programme team meetings. 


Quality culture 

A culture of improvement and open, self-critical evaluation is a prerequisite to 
effective internal review. In most colleges, progress has accelerated over the last 
three years. Staff in teaching and support areas play an active role in determining 
how well they are doing and what needs to improve further. In these colleges, there 
has been a shift from quality assurance to quality improvement. In the best colleges, 
staff are empowered to make decisions, often relating to key aspects of their 
programmes or service areas, and they are accountable for these outcomes. Their 
capacity has been developed and enhanced over time, and this has had a significant 
impact on the experiences of learners. The processes used to support accurate self- 
evaluation have been refined and improved and in many colleges these work well. 
Increasingly, colleges are seeking to align their self-evaluation reports against 
college-wide corporate aims and objectives. They recognise that actions derived 
from self-evaluation reporting can also be used as improvement targets for 
operational plans across other areas of provision. A number of colleges 
acknowledge that this is an area which they are targeting for improvement. 

All colleges successfully include support services in their self-evaluation activities. 
There is recognition by service departments that their function is to provide a high 
quality service for learners. In a number of colleges, these services have been 
significantly enhanced and improved through college self-evaluation arrangements. 
Service departments are, by definition, specialists in a particular role. Through 
discussions held with learners, learner surveys and through feedback received from 
staff, support service departments often quickly identify what is working well and 
what is required to continue to improve. Once identified, relevant action points are 
often quickly implemented resulting in improved services for both staff and learners. 


John Wheatley College: publicising self-evaluation 

At the time of writing, John Wheatley College is the only college in Scotland to 
publish its self-evaluation report annually on its website. The college believes that 
through this action the college demonstrates its commitment to openness and 
accountability and allows potential learners and other stakeholders to view for 
themselves the improvements being made at the college. 
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7. Conclusions 

The purpose of seif-evaluation is to improve outcomes for learners. Most colleges 
carry out self-evaluation conscientiously and well. However, despite this, across 
Scotland, overall attainment for full-time FE and HE learners, has not improved for 
the last three years. Clearly, there is still work to be done to identify why this is so 
and self-evaluation must have a key role to play in this. It is difficult to attribute 
improvements in performance specifically to robust self-evaluation. Nevertheless, 
there appears to be a correlation. In compiling this report, it was clear that, in 
general, those colleges which carry out robust self-evaluation benefit from PI 
outcomes, learning and teaching provision and support services which are not only 
good but improving. In these colleges, staff are talking openly and honestly with 
each other, are listening to their learners, and rigorously identifying patterns in 
learner performance which require changes in the approaches staff are taking. 
Managers in these colleges are supporting self-evaluation properly by monitoring 
outcomes and progress on actions, and providing useful benchmarking information, 
PI analysis and reporting pro forma. They are also using the outcomes of 
self-evaluation well to inform whole-college planning. Boards of Management know 
how well the college is doing and ensure that appropriate actions are in place to 
address areas of concern. 

Since the publication of Effective self-evaluation reporting in Scotland’s colleges 
(2007), self-evaluation has matured and improved. Colleges now have 
well-established and widely accepted quality assurance and improvement practices. 
Most of the recommendations for improvement contained in the 2007 report have 
been implemented. However, in more than a few colleges, self-evaluation is not 
consistently as effective as it could be and outcomes for learners in many areas are 
not improving. In these colleges, self-evaluation reporting continues to be 
insufficiently evaluative, does not analyse evidence rigorously enough and often fails 
to set specific actions and targets for improvement, particularly for learner progress 
and outcomes. Approaches to college-wide self-evaluation are insufficiently rigorous 
or robust resulting in managers having an overly optimistic view of their own 
performance and a belief that they do not have any main areas for development. As 
a result, targets for improvement are insufficiently challenging and do not ensure 
improved outcomes for learners. 

In general, colleges, including those whose self-evaluation is most effective, have 
not yet become fully proficient in evaluating learning and teaching, their core 
business. This vitally important area has direct impact on learners and colleges 
need to place the highest possible emphasis on it. In more than a few colleges, 
improvements are still required in approaches to equality and diversity. Often 
programme team action plans fail to address improvements for learners by defined 
equalities categories. In many colleges, self-evaluation reports omit reference to the 
promotion of equality and diversity through the introduction of refreshed teaching 
material or adjustments to teaching practices. In more than a few colleges, learner 
representative arrangements continue to be insufficiently effective. Overall, few 
colleges have established formal arrangements with schools and HE providers in 
order to review curriculum content and joint internal evaluation of programmes. 

Across Scotland, most colleges know their strengths and their areas for development 
well. They have well-developed practices in most areas and are aware of the 
aspects of their provision which they are striving to develop further and improve. 
However, improvements are still required by all colleges in some aspects of their 
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approaches to self-evaluation. In more than a few colleges, self-evaluation remains 
insufficiently robust and is not delivering improved outcomes for learners. 
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8. Recommendations 

The Scottish Funding Council should: 

• provide clearer guidance to colleges on the detail required in their annual 
reports; and 

• work with colleges to gain greater value from these reports. 

Scotland’s Colleges should: 

• continue to support colleges to improve their self-evaluation reporting. 

Colleges should: 

• ensure accurate and comprehensive approaches to college-wide self-evaluation; 

• compare PI outcomes with relevant external benchmarks; 

• ensure programme team action plans address improvements for learners by 
defined equalities categories and report on the promotion of equality and 
diversity; 

• ensure programme teams draw up appropriate targets for improvement, 
including approaches to learning and teaching; 

• ensure learner representation arrangements are effective; and 

• ensure internal review activities include contributions from representatives from 
local schools and receiving HE providers. 

Education Scotland should: 

• continue to monitor the progress made by colleges in taking forward the above 
recommendations. 
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Appendix 1 

Colleges visited in the fieldwork for this report 


• Aberdeen College 

• Angus College 

• Ayr College 

• Coatbridge College 

• Inverness College 

• John Wheatley College 

• Kilmarnock College 

• Newbattle College 
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Appendix 2 

Glossary of terms 


ACR Annual Course Report 

BoM Board of Management 

CL Curriculum Leader 

CPD Continuing Professional Development 

ESOL English for Speakers of Other Languages 

FE Further Education 

HE Higher Education 

HR Human Resources 

HMIE HM Inspectorate of Education 

ICT Information and Communication Technology 

PI Performance Indicator 

QELTM Quality and Equality in Learning and Teaching Materials 

Ql Quality Indicator 

SARU Student Unit Achievement Ratio 

SCR Successful Completion Ratio 

SFC Scottish Funding Council 

sparqs student participation in quality Scotland 

UHI University of the Highlands and Islands 
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